Application No.* 09/547,q^fc/ Response to Action 

Art Unit 2671 . dated 04/12/02 

through the imaging pipeline, manipulating the frame buffer to have stored therein color 
data of the 3D image which is to be rendered at each pixel based on the indication stored 
during the initial pass through the imaging pipeline. 



(New) The graphics system of claim 14, wherein said imaging pipeline further 
comprises: 

a stencil buffer containing stencil bits for each pixel in the display scene; and 
a stencil test module constructed and arranged to set said stencil bits to indicate 
which 3D image is closest toAhe viewpoint based on the results of the depth test. 



REMARKS 

1 . In response to the Office Action mailed April 12, 2002, Applicants respectfully 
request reconsideration. Claims 1-31 were originally presented for examination. All of 
the claims were rejected in the first Office Action. By the foregoing Amendments, claims 

2, 3, 6, 10, 13, 14, 18, 19, 21, 28, 30 and 31 have been amended. Claims 5, 9, 1 1, 12 and 
15-17 have been canceled without prejudice or disclaimer, and claims 32-37 have been 
added. Thus, with entry of this paper, claims 1-4, 6-8, 10, 13, 14 and 18-37 are pending 
in this application. Of these 30 claims, five claims (claims 1,18, 27, 28 and 3 1) are 
independent. These amendments are believed not to introduce new matter and their entry 
is respectfully requested. 

2. Based upon the above Amendments and following Remarks, Applicants 
respectfully request that all outstanding objections and rejections be reconsidered, and 
that they be withdrawn. 

Amendments To The Specification 

3. The title of the application has been amended to more clearly reflect Applicants' 
claimed invention. Entry is respectfully requested. 

Claim Rejections Under 35 U.S.C. §103 

4. The Examiner has rejected claims 1-35 under 35 U.S.C. 103(a) as being 
unpatentable over U.S. Patent No. 5,760,781 to Kaufman et al (hereinafter "Kaufman"). 
Specifically, the Examiner asserts that Kaufman teaches a two-dimensional (2D) imaging 



-6- 



Application No. 09/547,4 
Art Unit 2671 





Response to Action 
dated 04/12/02 



pipeline, and that it would have been obvious to a person of ordinary skill in the art at the 
time the invention was made to "configure Kaufman's system as claimed [in Applicants' 
independent claims]." The sole support provided by the Examiner in this obviousness 
rejection is "3D data [is converted] to 2D data to store in 2D buffers." (See, Office 
Action, pg. 2.) 

5. Applicants respectfully traverse this rejection. First, Kaufman does not disclose 
implementing a two-dimensional imaging pipeline as suggested by the Examiner. 
Second, the art of record neither teaches nor suggests modifying Kaufman to implement a 
2D imaging pipeline. Third, there is no teaching or suggestion on the art of record to 
composite separately-generated 3D images in any manner, let alone using a modified, 2D 
imaging pipeline as recited in Applicants' independent claims. Based on these facts, 
Applicants respectfully assert that a prima facia case for obviousness has not been made 
by the Examiner, requiring withdrawal of the outstanding Section 103 rejections. 

Kaufman Neither Teaches Nor Suggests a 2-DimensionaI Imaging Pipeline 

6. Kaufman is directed to a graphics system that performs real-time processing of 
voxels. To facilitate an understanding of the distinctions between Applicants' claimed 
invention and the system disclosed in Kaufman, Applicants offer the following 
information derived from pages 1-2 and 177-178 of "Introduction to Volume Rendering" 
by Barthold Lichtenbelt, Randy Crane and Shaz Naqvi, Prentice Hall PTR, 1998 
(hereinafter "Lichtenbelt"). The authors of this book were Hewlett-Packard employees at 
time of publication, and the book is commercially available from Prentice Hall as one of 
the "Hewlett-Packard professional books." A copy of these and other pages of 
Lichtenbelt cited below are enclosed herewith as Attachment 1 . Note that this 
information is being submitted for informational purposes only and not as prior art to 
Applicants' claimed invention. Accordingly, the excerpts are being submitted as an 
attachment to this paper rather than under 37 CFR § 1 .96. 

7. The operations performed by graphics systems to process voxels are collectively 
referred as voxel or volume rendering. Volume rendering is a method for directly 
rendering a volume data set, or volume, to show the characteristics of the interior of a 
solid object when displayed on a two-dimensional computer monitor. A volume data set 
is a three-dimensional array of data points referred to as volume elements, or voxels. A 
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voxel can be thought of as the three-dimensional analogue of a pixel; that is, a three- 
dimensional volume element representing a discrete sample of a volume in three- 
dimensional space. Voxels are commonly defined as sample points separated by a finite 
distance, with each voxel having a position and a value. The position of a voxel specifies 
the x,y,z coordinates of the center of the voxel in the 3D volume data set. The value of 
the voxel depends on its format. For example, a voxel can have an intensity value and an 
index value. Volume data sets can be generated by numerous means, but most commonly 
by some method of three-dimensional scanning or sampling and by numerical modeling. 
For example, a volume data set may be generated by magnetic resonance imaging (MRI), 
wherein the density of human or animal tissue is computed at each point of a 3D grid. A 
display of this information could indicate the boundaries of the various types of tissue, as 
indicated by density changes. (See, Lichtenbelt, pp. 1-2 and 177-178.) 

8. Kaufman discloses a graphics system that utilizes multiple 2D buffers and a high- 
speed bus to attain pipelined and parallel voxel rendering capabilities. (See, Kaufman, 
col. 2, In. 60- col. 3, In. 53.) Referring to Figure 2, Kaufman's volume visualization 
system operates by casting viewing rays 12 that originate at a pixel in the projection plane 
through a cubic frame buffer 22 along a selected viewing direction. In contrast to the 
then-conventional systems, Kaufman incorporates multiple 2D buffers in the manner 
disclosed in Figures 3 and 12 to increase the speed and efficiency of the volume rendering 
process. Referring to these illustrative embodiments, the Kaufman apparatuses 20 and 70 
include a cubic frame buffer 22 operationally connected to three 2D buffers 24. Each of 
the 2D buffers 24 are used as 2D voxel storage devices. The interconnecting mechanism 
between the buffers supports high-bandwidth transfer of beams of voxels from the cubic 
frame buffer 22 to the 2D buffers 24. (See, Kaufinan, col. 6, Ins. 20-63.) 

9. A volume rendering pipeline (that is, the pipeline disclosed in Kaufinan) is not the 
same as a 2D imaging pipeline or a 3D primitive-rendering pipeline. These rendering 
pipelines are briefly described below. The two-dimensional imaging pipeline for 
processing 2D images and the three-dimensional primitive rendering pipeline for 
processing primitives are described in Applicants' application: 
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There are two paths through a rendering pipeline, 
one for primitive data and one for pixel data. The former 
path processes three-dimensional (3D) primitive data to 
form a two-dimensional (2D) image for display on a 2D 
display screen. The latter path manipulates pixels, images 
and bitmaps of existing 2D images. The Cartesian 
coordinates of an image include X and Y coordinates 
corresponding to the pixel address in the horizontal and 
vertical dimensions of the display screen, and the Z 
coordinate which is orthogonal to the display screen and 
represents the distance from the location of the viewer, 
referred to as the viewpoint. The former path is referred to 
herein as a 3D pipeline due to the inclusion of depth 
information (Z coordinate data) in the primitive data. In 
contrast, the latter path is referred to as a 2D pipeline due to 
the absence of Z coordinate data from the 2D image data. 
This is because only the address (X,Y coordinate) and color 
(R,G,B,A) data is required to display a 2D image on a 2D 
display screen. 

(See, Applicants' application, pg. 3, In. 26-pg. 4, In. 8.) 

10. Neither of these pipelines are capable of processing voxels without substantial 
modifications being made to them. To process voxels, a graphics system must implement 
an additional pipeline, a volume rendering pipeline, that performs sequential operations 
on voxels. Such a volume rendering pipeline is described at pages 28-30 and 173-176 of 
Lichtenbelt. Copies of these pages are included in Attachment 1 . Briefly, a volume 
rendering pipeline includes segmentation, gradient computation, resampling, 
classification, shading and compositing stages. These voxel-processing stages perform 
operations that are not and cannot be performed in conventional 2D imaging or 3D 
primitive-rendering pipelines. Thus, there are three different types of rendering pipelines 
each having a specific structure to perform specific operations on a specific type of data: 
a 2D imaging or pixel pipeline renders 2D images; a 3D or geometric pipeline renders 
primitives; and a volume or voxel rendering pipeline renders voxels. There is no 
capability provided in one rendering pipeline to perform the operations of the other. 
Thus, appropriate hardware and software have to implemented in a graphics system to 
provide the desired rendering capability. 
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1 1 . Kaufman discloses performing voxel processing operations using a volume 
rendering pipeline that includes multiple 2D buffers and a high-speed bus. This 
arrangement provides benefits over then-conventional systems which, according to 
Kaufman, implemented a single 2D buffer and conventional bus. Applicants respectfully 
assert that the implementation of an additional 2D buffer and a high-speed bus in a 
volume rendering pipeline does not convert that volume rendering pipeline into a 2D 
rendering pipeline. In fact, the Kaufman volume rendering pipeline does not include any 
elements of a 2D imaging pipeline and, therefore, cannot process 2 -dimensional images. 
Nor do such modifications enable the Kaufman graphics system to composite separately- 
generated 3D images. Kaufman discloses only a volume rendering pipeline (or portion 
thereof) capable of rendering voxels, not images. Accordingly, Applicants respectfully 
assert that Kaufman does not disclose a 2D imaging pipeline as asserted by the Examiner. 

There is no motivation in the art of record to modify Kaufman as proposed 
in the office action 

12. Applicants assert that there is no suggestion in the art of record to modify 
Kaufman to utilize a two-dimensional graphics imaging pipeline. As noted, only volume 
rendering operations are performed in the Kaufman system. Neither Kaufman nor the 
other art of record provide the motivation to modify Kaufman to convert the volume 
rendering pipeline into a 2D imaging pipeline. Applicants further assert that the art of 
record provides no motivation whatsoever to composite separately-generated 3D images. 
In fact, the art of record is silent with regard to performing such operations or achieving 
such an objective. Thus, there is no teaching or suggestion in the art of record to modify 
Kaufman to render images, including 2-dimensional images which can be rendered by 
conventional 2D imaging pipelines, or 3 -dimensional images which can be rendered by a 
2D imaging pipeline as configured in Applicants' claimed invention. 

13. Thus, the Examiner has failed to provide any evidence, whether in the form of 
some teaching, suggestion, incentive or inference in Kaufman or other art of record, or in 
the form of generally available knowledge, that one having ordinary skill in the art would 
have been led to modify the relevant teachings of Kaufman in the proposed manner. This 
is because no such motivation exists in the applied references. Thus, the only conclusion 
that can be drawn, based on the record of this application, is that the suggestion forming 
the basis for the Examiner's otherwise factually unsupported conclusion must have come 
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from Applicants* own novel disclosure; that is, they are based on impermissible 
hindsight. It is too well-settled for citation that an applicant's own novel disclosure 
cannot be used to supply the teaching or suggestion that is missing from the known art. 
Furthermore, for the reasons set out above, Applicants assert that even if one of ordinary 
skill at the time of the invention were motivated to modify Kaufman as proposed by the 
Examiner, the resulting system would not contain nor would it have the advantages of 
Applicants' invention as recited in the independent claims. For at least these reasons, 
Applicants respectfully request that the rejection under Section 103 of the independent 
claims be reconsidered, and that they be withdrawn. 

14. Dependent claims 2-4, 6-8, 10, 12-14, 19-26 and 29-30 depend directly or 
indirectly from independent claims 1,18, 28, 28 and 31 and incorporate all of the subject 
matter of their respective independent claim. Furthermore, these dependent claims add 
additional subject matter which makes them independently patentable in and of 
themselves over the art of record. Accordingly, Applicants respectfully request that the 
rejection of the dependent claims be reconsidered, and that they be withdrawn. 

List of Attachments 

15. The following attachments form part of this Response and Amendment: 

Attachment 1 : Introduction to Volume Rendering" by Barthold Lichtenbelt, 
Randy Crane and Shaz Naqvi, Prentice Hall PTR, 1998, pgs. 
1-5, 28-30, and 173-178 

Attachment 2: Marked-Up Version of Claims Showing All Changes Made 

Attachment 3: Marked-Up Version of Specification Showing All Changes Made 
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16. In view of the foregoing Amendments and Remarks, this application should now 
be in condition for allowance. A notice to this effect is respectfully requested. 
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[Attachment 2 to the Amendment and Response filed in Response 
to the Office Action Dated April 12, 2002 
in U.S. Patent Application 09/547,065.1 

2. (Amended) The graphics system of claim 1 , wherein said 2D and 3D [two- 
dimensional] images [and said three-dimensional images] are represented by pixel data. 

3. (Amended) The graphics system of claim 1, wherein the graphics system comprises a 
rendering pipeline [including,] comprising: 

a geometric pipeline constructed and arranged to generate a two-dimensional 
image from one or more model views represented by primitive data; and 
said imaging pipeline. 

6. (Amended) The graphics system of claim [5,] 

wherein the graphics system further comprises a frame buffer for storing pixel 
data to be displayed on a display device, and 

wherein said 3D images to be composited comprise: 

a stored image stored in the frame buffer; and 

a next image to be composited with the stored image. 

10. (Amended) The graphics system of claim 14^ [9,] 

wherein [said pixel data of said stored image and said next image comprise color 
data] for each pixel, [and wherein] said imaging pipeline writes [color data] to [the] a 
frame buffer of the graphics system the color data of [one of] the 3D image that is closest 
to the viewpoint, [either said stored or said next image based on said indications of said 
selected image.] 

13. (Amended) The graphics system of claim l± [8,] wherein said 3D images are 
represented by pixel data comprising [of said next image includes] Z coordinate data, 
color data and X, Y coordinate data, wherein [an image compositing module of] said 
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imaging pipeline receives said Z coordinate data [and said color data] over a [color] data 
channel of the imaging pipeline configured to transfer data other than Z coordinate data. 
and receives said X,Y coordinate data over an address data channel. 



14. (Amended) The graphics system of claim L [13,] wherein said imaging pipeline 
[image compositing module] comprises: 

a depth buffer configured to store Z coordinate data for each pixel in a display 
scene; and 

a depth test module constructed and arranged to compare Z coordinate data of said 
3D images, [stored image and said Z coordinate data of said next image,] and to store in 
said depth buffer Z coordinate data of each pixel of the 3D [either said next or stored] 
image that is closest to scene viewpoint, [passed said depth test.] 

18. (Amended) A method for compositing 3D images in a 2D imaging pipeline to form a 
composited image comprising: 

storing in a frame buffer a stored 3D image including color data and Z coordinate 

data; 

processing in the 2D imaging pipeline Z coordinate data of a next 3D image to 
determine whether the stored or next 3D image is to be rendered at each pixel in the 
composited image; and 

replacing said stored color data with color data of said next 3D image for each 
pixel at which the next 3D image is to be rendered in the composited image., [at a pixel.] 

19. (Amended) The method of claim 18, wherein said processing Z coordinate data 
comprises: 

transferring Z coordinate data of the next image through an available data channel 
of imaging pipeline; 

depth testing the stored and next images; 

updating a depth buffer as necessary to store [with] Z coordinate data of a closest 
image; and 

recording an indication of which 3D image is the closest image. 
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21. (Amended) The method of claim 18, wherein the imaging pipeline consists of a color 
data channel and an address data channel, and wherein replacing the stored color data 
comprises: 

receiving over [routing selectively] the color data channel [to a depth test module] 
the color data of the next 3D image, and 

storing the color data of the next 3D image when the recording step records an 
indication that the next 3D image is the closest 3D image . 

28. (Amended) A graphics system comprising [an] a two-dimensional imaging pipeline 
configured to manipulate two-dimensional (2D) images and to composite separately^ 
generated stored three-dimensional (3D) image stored in a frame buffer, and a next 3D 
image, [images,] comprising, 

a color data channel adapted to receive Z coordinate data and color data of a next 
3D image; 

an image compositing module configured to perform [depth testing and stencil 
testing on specific components of next image, wherein said specific components are 
separately and sequentially processed by the imaging pipeline such that stencil testing of 
any one pixel is performed after depth testing for all pixels,] 

[wherein said] a depth test [is performed] to [first] determine which [of the stored 
and next] 3D image is to be rendered at each pixel based on Z coordinate data of the next 
image received over the color data channel, and Z coordinate data of the stored 3D image, 
and to store the Z coordinate data of the 3D image to be rendered at that pixel in a depth 
buffer, [two images,] and [wherein] a [said] stencil test [is performed using the results of 
said depth test] to form a stencil mask identifying which 3D image is the image that is to 
be rendered at each pixel , 

wherein the imaging pipeline, in response to receipt of color data over the color 
data channel, updates a color buffer to have stored therein color data of the 3D image to 
be rendered at each pixel of the composite image, [first determination is performed on Z 
coordinate data of the stored and next images while said second determination is 
performed on color data of the next image.] 

30. (Amended) The graphics system of claim 28, wherein the graphics system comprises 
a rendering pipeline comprising: [including,] 
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a geometric pipeline constructed and arranged to create a two-dimensional image 
from [one or more model views defined by] primitive data; and 
said imaging pipeline. 

3 1 . (Amended) A graphics system comprising a graphics application controlling a 2D 
imaging pipeline to composite a first 3D image and a second 3D image generated 
separately than the first image, wherein the imaging pipeline processes Z coordinate data 
of the images to determine, for each pixel in the composited image, which of the first or 
second 3D image is closest to a viewpoint, and stores [causes] color data of the closest 3D 
image [to be stored] in a frame buffer for [subsequent] rendering on a display device. 
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Marked Up Version of Specification 
Showing All Changes Made 



[Attachment 3 to the Amendment and Response filed in Response 
to the Office Action Dated April 12, 2002 
in U.S. Patent Application 09/547,065.] 



The Title on page L lines 1-2 



[COMPOSITING SEPARATELY GENERATED 3D IMAGES 
USING A GRAPHICS RENDERING PIPELINE] 

COMPOSITING SEPARATELY-GENERATED 
THREE-DIMENSIONAL IMAGES 
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